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SYSTEM A^/IETHOD FOR PROVIDING SE^^TY MECHANISMS 
FOR SECURING NETWORK COMMUNICATION 

BACKGROUND OF THE INVENTION 

1. Field of the Invention : 

5 The present invention relates to computer networks and network security, and in 

particular, to systems and methods for providing security mechanisms for securing 
manageability in a computer network. 

2. Related Art : 

Computer networks in business enterprises, such as a local area network (LAN), 
10 wide area network (WAN) or other Ethernet-based systems facilitate communication 
among computer workstations. With the recent evolution of networking and Internet 
communications, computer networks have become more open to the world. While this 
certainly speeds business operations, it brings with it other perils. Having computer 
networks more open to the world can often leave data and networks traffic open to 
15 unintended access. An outsider may install and use a program to monitor the network 
traffic, alter or modify data streams in transit, or steal an identity to gain unauthorized 
access into a network. Therefore, a secure environment requires protection at the 
network level. 

A typical LAN couples together one, or a relatively small number of, server 
20 systems and potentially large number of client systems. Network traffic communicated 
between any two systems is in the form of data packets and utilizes protocols regulating 
the way the data packets are transmitted between the two systems. Many security 
protocols are provided for securing network traffic. In the case of a LAN, Internet 



Protocol Security (110) technology has emerged as thfel security protocol of 
choice. IPSec allows business enterprises to add internal LAN protection, building 
communications security into the data packet itself and securing client/server 
communications. IPSec operates at the network layer of the protocol stack, i.e., Layer 
3 in the Open System Interconnection (OSI) model, and can be used to provide three 
different types of protection: authentication, integrity and encryption. 

IPSec may be applied in many instances. For example, the server system may 
be a remote management station wishing to communicate certain management traffic to 
a client system. The remote management station would utilize a management IP based 
protocol, such as IPSec, to initiate certain management operations on the client system. 
This is especially true when the client system becomes non-operational, e.g., when the 
client system is in a pre-boot state, a hung state, or a reset state. In this case, the 
remote management station would want to send out management commands to try and 
get the client system back to an operational state. For example, the management 
commands may include reset, reboot, power down, or power up. These heavy-duty 
control commands, which can reset or reboot any client systems connected in a 
network, need to be securely communicated. When a client system is non-operational 
and another system is toying to manage the client system, care must be taken to make 
sure that the other system is indeed a management station that the client system trusts. 

A typical communication security protocol between two systems has two phases, 
in the first phase, typically referred to by the name "key exchange", the systems 
authenticate each other as well as negotiate and agree upon exact parameters and 
keys to be used to secure subsequent network traffic. The parameters and keys to be 
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used represent the £ts obtained after carrying out th^exchange processes, and 
are often referred to as security association (SA). The SA contains settings like 
policies and the extent of the strength of the security that is employed on a connection 
basis. In the second phase, network traffic is secured based on the results obtained in 
5 the first phase. 

The typical security protocols like the key exchange processes are fairly complex 
and require many exchanges and computationally intensive operations. This means 
they do not work well when the operating system (OS) of the client system is absent, 
i.e., when the client system is non-operational. Although existing security mechanisms, 
10 such as those utilizing IPSec and Internet Key Exchange (IKE), are able to secure 
network traffic when both the client system and server system are operational, they 
cannot secure network traffic when the OS of the client system is non-operational or 
absent. There is a need for a method to securely communicate network traffic, 
regardless of the state of the client system under consideration. 
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SUMMARY ^ 

Embodiments of the present invention are directed to addressing the 
afo remen,ioned drawbacKs associated w*h providing security mechanisms tor securing 
W«c communicated from one system to another system, even when one o, the 
systems is non-operational. An embodiment of the present invention is directed to a 
system and method of providing securfty mechanisms for securing traffic 
communication between a sen,er system and a Cien, system regardless o, the state of 
, h e client system under consideration. First, the client system is poiied to determine 
wh e,her it is in an operational state. As soon as the *n< system enters the 
operation, state, Key exchange processes are initiated and executed between the 
server system and the client system. At the end of the key exchange processes, the 
,esu„s o, the Key exchange processes are stored into the client system. The traffic 

the stored results in the client system. In order to maintain a highly secured 
environment, the stored results in the client system are periodic* refreshed and 
updated wnh newfy obtained resuKs by executing a second set of Key exchange 
processes between the server system and the client system. 

However, by inhibiting the stored resutts in the den, system from being updated 
u „ ti , a successful execution o.the second set of Key exchange processes is actually 
„ carried out, the system ensures that the traffic is securely communicated even ,f the 
Cientsystem becomes non-operational. ,n .his case, the system will use the previously 
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co— ion. Traffic communication between ,#ver system and «he client 
system can be secured at all times, even in a non-operation state. 
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BRIEF DESCRIPTlflfOF THE FIGURES 

Figure 1 shows a local area network coupling a server system and a client 
system according to an embodiment of the invention. 

Figure 2 shows in more detail an embodiment of the client system according to 

the embodiment of Figure 1 . 

Figure 3 illustrates processes for carrying out securfty mechanisms according to 

an embodiment of the invention. 

Figure 4 illustrates processes for updating the results of key exchange processes 
at a server system according to an embodiment of the invention. 

Figure 5 illustrates processes for updating the results of key exchange processes 
at a client system according to an embodiment of the invention. 

Figure 6 shows a table illustrating the relationship between a sewer system and 
a client system during different state transitions according to an embodiment of the 



invention. 
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DETAILED DESCR^fcoN 

Embodiments of the present invention are directed to a system and method of 
providing security mechanisms for securing traffic communicated from one system to 
another system independently of whether the latter system is running normally, or is in a 
5 non-operational state. The systems are preferably a sewer system and a client system, 
each system containing an operating system and being connected through a computer 
network. The sewer system preferably determines whether the client system has 
entered fully operational state. Once the client system enters the fully operational 
state, key exchange processes are Mated between the two systems to obtain security 
,0 parameters for use in securing traffic communication between the two systems. The 
security parameters, calfcd SA, are the resuKs acquired a. the end of the key exchange 
processes. After the key exchange processes are over, the SA is stored in the client 
system. To maintain a highly secured environment, the server system periodically 
retries the SA by periodically executing another set of key exchange processes, and 
,5 communicates the newly obtained SA to the Cien. system for storing them in the client 
system in place of the original security association. However, the SA is inhibfted from 
being updated in the client system until the server system is successful in completely 
executing another set of key exchange processes. The traffic communicate is then 
secured based on whichever SA is stored in the client system. Depending on whether 
20 the other set of key exchange process is successful, the traffic communication is 
secured on the basis of either the original SA or the newly obtained SA. 

Figure 1 shows a server system 2 and a plurality of client systems 3 coupled in a 
LAN 1 according to an embodiment of the invention. The typical arrangement of an 



20193076V1 



environment on a includes one, or a revive sm#mber of, .he server 

s y s t em2andapo,eTl,Vlar g enu mb ero,*n t svs,ems, In other em*—, the 

other networK having an ,nternet Protoooi (IP) based protocol for transmrtting data 

5 packets. 

Figure 2 shows in more detail an embodiment of the client system 3 in Figure 
The Cien, system 3 includes a network device 4 and a processor 5 for executing 
computer instruction, According to an embodiment, the processor 5 of the Cent 
system 3 includes a centra, processing unit (CPU) and random access memo* (RAM) 

NT 4 0 operating system and application programs compatibie wKh these operat,ng 
systems, in other embodiments, combinations ofdifferen. operating systems and 
different application programs may be Implemented in the client system 3. According to 
an embodiment, a remote server system 2 manages the dien. system 3 by 
15 communicating to client system 3 the management functions reared to be performed. 
ln the embodiment, the server system utilizes a management IP based protocol in 
carrying out certain ^diagnostic operations on a client system 3. In the 
embodiment, securKy mechanisms, independent of the state the client system 3 ,s ,n, 
are provided, so that management IP based protocol packets can be securely 

20 communicatedtromtheremote ystem 2 to the client system 3 a, all times. Th,s 

way the security mechanisms ensure that the management IP based protocol packets 
originate from a trusted server system 2. For example, the security mechanisms may 
be used to secure remote management and control protocol (RMCP) packets using 
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IPSec through the stages in the IKe cycle of the s# system 2 and the client 



system 3. 

According to an embodiment, the control/diagnostic operations are performed 
through the network device 4 in the client system 3. The management IP based 
protocol is usually fairly simple and uses a particular user datagram protoco, (UDP) port 
to communicate the management traffic. With the security mechanisms, the remote 
server system 2 securely communicates the management traffic to the network device 
4. After the network device 4 intercepts the management traffic, it triggers certain 
control/diagnostic operations, such as reboot, on the client system 3. In other 
embodiments, the network device 4 may be considered as the client system, in which 
, h e security mechanisms are provided to secure communication between a sender 

system and a network device. 

in a preferred embodiment, the security mechanisms operate mainly at the 
network layer of the seven-layer protocol stack, i.e.. Layer 3 in the OSI stack. The 
network layer embodiment is preferred because of the existence of the IPSec. In other 
embodiments, the IP based protoco, used by the securrty mechanisms along wfth the IP 
based protoco, packets may use hfcher layers o, the OS, stack. Moreover, the secutfy 
mechanisms may be imptemented in the data-iink layer, i.e., Layer 2 in the OS, stack. 
For example, the server system may dial in to the client system, and send traffic 
, communication at the point to point protocol (PPP) layer. 

Figure 3 illustrates processes for carrying out security mechanisms according to 
an embodiment of the invention. ,n step 10. security mechanisms are Mated by a 
server system to detect whether a Cent system is operational. If the Cent system is 
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operation*, Key ex^e processes are executed betw^he server system and the 
client system in step 11. a, the end o. which the resuits o.the Key exchange process, or 
SA are obtained. According to one embodiment, Key exchange processes are earned 
out utilizing IKE, which supports the verification of identities. IKE hybrid protocol. 
5 and b purpose is to negotiate and provide authenticated Keying materia, for IPSec SAs 
M are used for Authentication Header (AH) and Encapsulating Security Payload 
(ESP) processing. After the SA is obtained after the execution of me Key exchange 
processes in step 1 1 , the SA is stored in the client system in step 12. In one 
embodiment, the SA is stored in a networK device that is part of the client system. For 
,„ example, the SA is stored in an Ethernet device. In other embodiments, the networK 
device ttself is consktered as me client system, and the SA stored in a component 
par, of the network device. For example, the SA is stored in a coprocessor connected 
,o an Ethernet device or an EEPROM/flash that is part of an Ethernet device. 
After the SA is stored, the server system initiates refreshing of the SA by 
15 executing another se, of key exchange processes based on a SA refresh timer. 
Periodically refreshing of the SA to generate new SA is required to provide an 
environment with more security and protection. Steps 1 3 and 14 may be viewed as 
mechanisms for MM* the SA in the client system from being updated until there ,s a 
successful completion o, the SA refresh in the server system. The SA in the client 
20 system is updated on* after a successful refresh is completed, in step 15, the traffic 
communication is secured based on the resuits stored in the client system. Thus, 
depending on whether another set of Key exchange processes between the server 
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system and the clie^stem is successful, the traffic co^hication is secured either 
with the SA stored in step 12 or the updated SA in step 14. 

According to one embodiment, when the network device in the client system 
receives a secured packet through the network, it processes the packet to validate 
security of the packet based on whatever SA is stored in the client system. This may 
involve, for example, crypotographic hash, decryption, and other processes such as 
checks for replay attacks. After successful processing of packet for security, the 
network device forwards the packet to a processor in the client system, allowing normal 
remote management processing. Similarly, any response generated by the 
management module to be sent to the server system is also processed for security 
based on the prior SA negotiation. 

Figure 4 illustrates processes for updating SA at a server system according to an 
embodiment of the invention. In step 20, the sender system determines whether the 
client system is operational. If the client system is non-operational, previously 
negotiated SA, which is the same SA as the one stored in the client system, is used to 
secure traffic communication until the client becomes operational. If the client system is 
operational, the server system determines whether SA should be refreshed based on a 
SA refresh timer, according to an embodiment of the invention. In step 21 , the server 
system asks whether the SA refresh timer has timed out. If it has not, the previously 
negotiated SA is used to secure traffic communication until the SA refresh timer has 
timed out. On the other hand, if the timer has timed out, the server system initiates a 
SA refresh in step 22 by carrying out another set of key exchange processes. 
According to an embodiment, the time between consecutive refresh of SA is set to be 
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signHicanfiy smalleA the «me of ft. SA, where th^rence must be a, leas, as 
,arge as the maximum allowed down time before the client system is managed, .n a 
conserve design, one may choose a very large lifetime for SA. while refreshing SA 
faidy frequently. For example, the Wefime for SA may be in years, while refreshing the 

5 SA may be in hours or minutes. 

in step 23. the server system determines whether i, has received a "SA is ready 
f0 r use- signal from the client system. If such signal is no. received, the previously 
negotiated SA is used to secure traffic communication in step 30. If such signal has 
been received, the server system checks whether the other set of key exchange is 
10 successfully completed in step 24. .f there is an unsuccessful complex of the SA 
refresh, the previousfy negotiated SA is used to secure traffic communication in step 30. 
Hthere is a successfcl completion of the SA refresh, the server system sends an 
acknowledgement signal to the client system and waKs for a confirmation signal from 
the client system confirming the receipt of the acknowledgement signal (steps 25 and 
15 26, If confirmation is not received from the *n« system, the previousfy negotiated SA 
is used to secure traffic communication in step 30. On the other hand, » confirmation ,s 
received, the newly refreshed SA is used to secure traffic communicafion in step 27. 
' Figure 5 illustrates processes for updating SA at a client system according to an 
embodiment of the invention. In step 40, the client system determines whether there is 
20 new SA available for storage. If new SA is not available, then the client system uses 
me previously negotiated SA that is stored in the client system to secure traffic 
communication unfil the new SA is available. If a new SA is availabfc. the new SA is 
stored in the client system in step 41 . According to an embodiment, the new SA is 
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stored in hardware client system, preferably a net^device. In step 42, the 
client system indicates that the new SA is ready for use by sending the "SA is ready for 
use" signal to the server system. In step 43, the client system waits for the 
acknowledgement signal from the server system. If the acknowledgement signal is not 
5 received, then the client system restores the previously negotiated SA and uses it to 
secure the traffic communication in step 52. If the acknowledgement is received, the 
client system sends the confirmation signal to the server client in step 44. In step 45, 
the new SA is used to secure the traffic communication. 

According to one embodiment, the client system provides configuration options 
10 such that it can be managed with or without security. When a new client system is 
installed and does not have any OS present, the key exchange processes cannot be 
executed. The new client system is managed without security through configuration 
options by using some non-volatile storage such as an EEPROM or a register. By 
setting appropriate bits on the non-volatile storage, securing traffic is controlled. In 
15 another embodiment, a server system fails and becomes non-operational while the 
client system becomes non-operational. As a result, SAs on the server system are lost 
and no longer exist on the server system. In this case, the server system implements a 
persistent store for storing SAs that are in use with a plurality of client systems. The 
previously negotiated SAs are then easily restored. The persistent store may be some 
20 non-volatile storage such as an EEPROM. 

Figure 6 shows a table illustrating the relationship between a server system and 
a client system during different state transitions according to an embodiment. Each row 
represents a transition state for the client system and the server system, and describes 
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«H column describes attributes of the transition states. 

* m « r thP client system could be in 
.ntbeembodimentthestatesthesewersystemortheCent 

are "OS up," "OS Hung," "Pre-boot," "OS suspend," "Cold boot," and "Any state. OS 

« , hoot having a pre-established securty context, but IKE 
system hangs after a successful boot, havmg a p tnavinaap re- 

uole "Pre-boot" represents a state when a system ,s reset, hav.ng pre 
being unliable. Pre bo . Cold . boo t" represents a state 

0 established security context, but IKE being u ^ . 0Ssuspe nd" 

wh enasys«e m co m esoutco W ,havingnosecuri^con,ext.ore,yupon. OSsusp 

Cenlas.te W henasystemis,empora ri ,sus P ended,e,,toconse„e P o W er 

z^^^^^^^x 

« - A nv stole" refers to any of the above mentioned states. 

+ AH, Destination port - 298h, other parameters - Wldcard, 

« te m is "OS up" and the client system transits to an OS up 
20 casewheretheserversystemrsOSupa 

sta ,e.Sincebo,syste m sare"OSup;theyareful, opera„ona, 
the „stemt— s,oan"OSu P "s,ate,aSArefreshisem P loyed,andne W SA 
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would be updated i£ client system and used to secu£ traffic communication if 
there is a successful completion of the SA refresh. This ensures that there are no 
sequence number synchronization issues that have to be addressed. 

Row 3 indicates the case where the server system is "OS up" and the client 
system transitions to an "OS hung" state. Row 4 indicates the case where the server 
system is "OS up" and the client system transitions to a "Pre-boot" state. Row 5 
indicates the case where the server system is "OS up" and the client system transitions 
to a "OS suspend" state. When the client system undergoes a transition from an "OS 
up" state to a "OS hung," "Pre-boot" or "OS suspend" state, the previously negotiated 
SA stored in the client system is inhibited from being updated until a successful 
execution of another set of key exchange processes between the server system and the 
client system. In this case, the previously negotiated SA is still operational because it is 
stored in the client system. According to an embodiment, the server system stops 
renegotiating new SA by not completing a SA refresh since IKE does not exist anymore, 
and the server system continues to use previously negotiated SA to secure traffic 
communication. In other embodiments, selective communication is permitted on an 
insecure port, which can be enabled based on lack of communication with the server. 

Row 6 indicates the case where the server system is "OS up" and the client 
system is "Cold-boot." When the client system is in such state, the client system does 
not have any security information. For example, a new system is in "Cold-boof when it 
is installed and does not have any OS present, and therefore, the key exchange part of 
the security protocol cannot be executed. According to one embodiment, the client 
system provides a configuration option such that it can be managed with or without 
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security. ,n this casfe client system sends traffic coition in the dear. 
prefe rab,y with restrictions on the data, to the server system. In order to maKe it easier 
on the server, the traffic communication sent in clear may be sen, on a dKferent UDP 

port. 

row 7 indicates the case where the server system is "OS hung- and the client 
system is "Any state." Under this condition the client system is unmanageable by the 
server system because the server system is non-operational. A fault tolerant system is 
implemented for this case, wherein the fault tolerant system switches the control from 
the "OS hung' sever to an "OS up" secondary server. 

row 8 indicates the case where the se^er system is "Cold-boot' and the client 
system is "OS Hung or Pre-boot." Since the sen,er system is in "Cold-boot." the server 
system does not have any security information. According to one embodiment, the 
dient system provides a configuration op«on such that it can be managed with or 
without secure In this case, the client system takes unilateral action and 
communicates on an insecure port. The client system sends traffic communication ,n 
the clear, preferably with restrictions on the data, to the server system. 

row 9 indicates the case where *e server system is "Cold-boot" and the client 
system is "Cold-boot." Since both systems are in "Cold-boot," neither one has any 
security information. In this case, the traffic communication is in the dear on the 
0 insecure port until such time that a security context can be established. 

While the description above refers to particular embodiments of the present 
inv en«on. i, will be understood that many mod*ca«ons may be made without departing 
from the spirn thereof. The accompanying claims are intended to cover such 
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modifications as wo£all within the true scope and spi^he present invention. The 
presently disclosed embodiments are therefore to be considered in all respects as 
illustrative and not restrictive, the scope of the invention being indicated by the 
appended claims, rather than the foregoing description, and all changes which come 
within the.meaning and range of equivalency of the claims are therefore intended to be 
embraced therein. 
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